1. Field
The present disclosure relates to a video deblurring method based on a layered blur model and a recording medium and device for performing the same, and more particularly, to a video deblurring method for accurately restoring object boundaries as well as each layer with the consideration of interaction between the layers, and a recording medium and device for performing the same.
2. Description of the Related Art
Layered generative blur model is frequently used in the motion deblurring field to handle locally varying blurs, which is caused by object motion or depth variation in a scene. However, conventional layered blur models have a limitation in representing the layer interactions occurring at occlusion boundaries. Thereby an error may occur in deblurring results at object boundaries.
Motion blur is one of the most common artifacts in video. There have been extensive studies to remove the motion blur, and they are roughly categorized into spatially-invariant and spatially-variant configurations. Early works on spatially-invariant blur achieved some success in single-image deblurring, and these techniques were extended to video deblurring.
However, the spatially-invariant blur model cannot deal with a rotational camera motion, which is a significant and common component in practical scenarios. To overcome this limitation, some researchers parameterized the blur as a possible camera motions in 3D, and this approach is applied to single-image and video. Although these methods solved spatially-variant motion blur in some extent, they were limited to the smoothly-varying blur which represents a global motion of the camera in single depth image.
In the case of abruptly-varying blur, generated by depth variation or object motion, the blur cannot be represented by a simple homography, but the blur kernel of each segment should be estimated separately. To address this problem, Lee and Lee proposed methods that estimate depth maps from blurry images based on the commutative property of convolution. Given this estimated depth maps, they obtained spatially varying blur kernel that can cope with depth variation in a static scene.
For a dynamic scene including object motion, Cho et al. segmented an image into regions of homogeneous motions by estimating the motion simultaneously, and obtained corresponding segment-wise blur kernel for restoring the image. Also, Cho et al. used patch-based synthesis for deblurring by detecting and interpolating proper sharp patches at nearby frame.
Although these methods solved abruptly-varying blur by estimating the blur of each segment separately, they did not consider a very important problem seriously, that is ‘the occlusion problem during a blur process’. Occlusion in motion causes an interaction between the segments because a pixel value around object boundaries becomes a weighted sum of pixels from multiple segments, and it makes the deblurring problem even more difficult.
To handle this interaction, several researchers proposed layered blur models that express boundaries of moving objects as a linear combinations of the blurred foreground and the background. However, their methods assumed the background to be static and only foreground motion was modeled, so there were limitations in representing general blurs.
To overcome these limitations, Wulff and Black recently proposed a method that explicitly models a blur image as a composition of individually blurred foreground and background. This generative blur model expresses not only the motion of each layer but also the interaction between the layers caused by the occlusion during the blur process. By making use of this model, they could handle motion blurred videos even when both layers are dynamic while considering the occlusion.
However, their blur model is not generally true but only valid for some specific physical situations. For example, in the case of blurry images caused by the situation that both the hand and the background are moving to the right-side at the same speed, a blur is seen, in actual occurrence, as an overlap of images the camera sees while the shutter is open.
Accordingly, since some regions in the background are occluded by the hand all the time while the shutter is open, it should not be exposed in the blurry image. However, since in synthesized blurry images by conventional models, the background occluded by the hand is exposed in some instances, it shows that the model violates the actual data-acquisition process. This error in blur generative model can cause artifacts not only in synthesized blurry images but also in deblurring results. Therefore, there is the demand for a robust video deblurring model that allows accurate deblurring even in general situations.